{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2e8b098f216b80e5",
   "metadata": {},
   "source": [
    "## 位置编码\n",
    "数学公式\n",
    "$$PE(pos, 2i) = sin(pos / 10000^{2i / d})$$\n",
    "$$PE(pos, 2i+1) = cos(pos / 10000^{2i / d})$$\n",
    "\n",
    "PE为最终的位置编码矩阵，矩阵形状为 (max_len, d_model)，一行就是一个位置的编码，一行的维度为d，d就是论文中的d_model\n",
    "\n",
    "pos表序列中的所有位置，长度为max_len，表示PE中的某一行，比如一个seq的长度为256，那么pos的大小就是256，也就是PE的行数\n",
    "\n",
    "i表示每一行中的某个列，比如d_model为512，那么i的范围就是[0, 511]，也就是PE的列数，PE有256行，512列\n",
    "\n",
    "2i / d，就表示把矩阵中的某一列乘以2然后除以d\n",
    "\n",
    "最终的PE矩阵和输入i的维度是一样的，都是一行表示一个token，一行的长度表示位置编码和词向量，位置编码和词向量长度一样，这就就能相加，从而在计算注意力机制时就能考虑token的位置信息了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "16f1a7d120fdeef8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-05-25T03:22:47.528046Z",
     "start_time": "2025-05-25T03:22:47.522810Z"
    }
   },
   "outputs": [],
   "source": [
    "max_len = 8  # 最大序列长度\n",
    "d_model = 16  # 模型维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "be4fc4bdd93c73d4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-05-25T03:22:49.974663Z",
     "start_time": "2025-05-25T03:22:49.967224Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# 初始化位置编码矩阵\n",
    "# 每一行表示一个位置的编码，也就是一个token对应的位置信息，编码的长度为d_model，也就是注意力中输入i1的对应的词向量的大小\n",
    "pe = torch.zeros(max_len, d_model)\n",
    "pe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2d1ed94fd5d28bad",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-05-25T03:22:52.498331Z",
     "start_time": "2025-05-25T03:22:52.491750Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.],\n",
       "        [1.],\n",
       "        [2.],\n",
       "        [3.],\n",
       "        [4.],\n",
       "        [5.],\n",
       "        [6.],\n",
       "        [7.]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 生成位置索引 [0, 1, 2, ..., max_len-1]\n",
    "pos = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
    "pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b065ff6d18ecd049",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-05-25T03:22:54.787410Z",
     "start_time": "2025-05-25T03:22:54.780319Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,\n",
       "          0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,\n",
       "          0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,\n",
       "          0.0000e+00],\n",
       "        [ 8.4147e-01,  0.0000e+00,  9.9833e-02,  0.0000e+00,  9.9998e-03,\n",
       "          0.0000e+00,  1.0000e-03,  0.0000e+00,  1.0000e-04,  0.0000e+00,\n",
       "          1.0000e-05,  0.0000e+00,  1.0000e-06,  0.0000e+00,  1.0000e-07,\n",
       "          0.0000e+00],\n",
       "        [ 9.0930e-01,  0.0000e+00,  1.9867e-01,  0.0000e+00,  1.9999e-02,\n",
       "          0.0000e+00,  2.0000e-03,  0.0000e+00,  2.0000e-04,  0.0000e+00,\n",
       "          2.0000e-05,  0.0000e+00,  2.0000e-06,  0.0000e+00,  2.0000e-07,\n",
       "          0.0000e+00],\n",
       "        [ 1.4112e-01,  0.0000e+00,  2.9552e-01,  0.0000e+00,  2.9995e-02,\n",
       "          0.0000e+00,  3.0000e-03,  0.0000e+00,  3.0000e-04,  0.0000e+00,\n",
       "          3.0000e-05,  0.0000e+00,  3.0000e-06,  0.0000e+00,  3.0000e-07,\n",
       "          0.0000e+00],\n",
       "        [-7.5680e-01,  0.0000e+00,  3.8942e-01,  0.0000e+00,  3.9989e-02,\n",
       "          0.0000e+00,  4.0000e-03,  0.0000e+00,  4.0000e-04,  0.0000e+00,\n",
       "          4.0000e-05,  0.0000e+00,  4.0000e-06,  0.0000e+00,  4.0000e-07,\n",
       "          0.0000e+00],\n",
       "        [-9.5892e-01,  0.0000e+00,  4.7943e-01,  0.0000e+00,  4.9979e-02,\n",
       "          0.0000e+00,  5.0000e-03,  0.0000e+00,  5.0000e-04,  0.0000e+00,\n",
       "          5.0000e-05,  0.0000e+00,  5.0000e-06,  0.0000e+00,  5.0000e-07,\n",
       "          0.0000e+00],\n",
       "        [-2.7942e-01,  0.0000e+00,  5.6464e-01,  0.0000e+00,  5.9964e-02,\n",
       "          0.0000e+00,  6.0000e-03,  0.0000e+00,  6.0000e-04,  0.0000e+00,\n",
       "          6.0000e-05,  0.0000e+00,  6.0000e-06,  0.0000e+00,  6.0000e-07,\n",
       "          0.0000e+00],\n",
       "        [ 6.5699e-01,  0.0000e+00,  6.4422e-01,  0.0000e+00,  6.9943e-02,\n",
       "          0.0000e+00,  6.9999e-03,  0.0000e+00,  7.0000e-04,  0.0000e+00,\n",
       "          7.0000e-05,  0.0000e+00,  7.0000e-06,  0.0000e+00,  7.0000e-07,\n",
       "          0.0000e+00]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# 偶数列\n",
    "pe[:, 0::2] = torch.sin(pos / (10000 ** (2 * torch.arange(0, d_model, 2) / d_model)))\n",
    "pe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-05-25T03:23:11.288045Z",
     "start_time": "2025-05-25T03:23:11.281190Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000e+00,  1.0000e+00,  0.0000e+00,  1.0000e+00,  0.0000e+00,\n",
       "          1.0000e+00,  0.0000e+00,  1.0000e+00,  0.0000e+00,  1.0000e+00,\n",
       "          0.0000e+00,  1.0000e+00,  0.0000e+00,  1.0000e+00,  0.0000e+00,\n",
       "          1.0000e+00],\n",
       "        [ 8.4147e-01,  9.5042e-01,  9.9833e-02,  9.9950e-01,  9.9998e-03,\n",
       "          9.9999e-01,  1.0000e-03,  1.0000e+00,  1.0000e-04,  1.0000e+00,\n",
       "          1.0000e-05,  1.0000e+00,  1.0000e-06,  1.0000e+00,  1.0000e-07,\n",
       "          1.0000e+00],\n",
       "        [ 9.0930e-01,  8.0658e-01,  1.9867e-01,  9.9800e-01,  1.9999e-02,\n",
       "          9.9998e-01,  2.0000e-03,  1.0000e+00,  2.0000e-04,  1.0000e+00,\n",
       "          2.0000e-05,  1.0000e+00,  2.0000e-06,  1.0000e+00,  2.0000e-07,\n",
       "          1.0000e+00],\n",
       "        [ 1.4112e-01,  5.8275e-01,  2.9552e-01,  9.9550e-01,  2.9995e-02,\n",
       "          9.9995e-01,  3.0000e-03,  1.0000e+00,  3.0000e-04,  1.0000e+00,\n",
       "          3.0000e-05,  1.0000e+00,  3.0000e-06,  1.0000e+00,  3.0000e-07,\n",
       "          1.0000e+00],\n",
       "        [-7.5680e-01,  3.0114e-01,  3.8942e-01,  9.9201e-01,  3.9989e-02,\n",
       "          9.9992e-01,  4.0000e-03,  1.0000e+00,  4.0000e-04,  1.0000e+00,\n",
       "          4.0000e-05,  1.0000e+00,  4.0000e-06,  1.0000e+00,  4.0000e-07,\n",
       "          1.0000e+00],\n",
       "        [-9.5892e-01, -1.0342e-02,  4.7943e-01,  9.8753e-01,  4.9979e-02,\n",
       "          9.9988e-01,  5.0000e-03,  1.0000e+00,  5.0000e-04,  1.0000e+00,\n",
       "          5.0000e-05,  1.0000e+00,  5.0000e-06,  1.0000e+00,  5.0000e-07,\n",
       "          1.0000e+00],\n",
       "        [-2.7942e-01, -3.2080e-01,  5.6464e-01,  9.8205e-01,  5.9964e-02,\n",
       "          9.9982e-01,  6.0000e-03,  1.0000e+00,  6.0000e-04,  1.0000e+00,\n",
       "          6.0000e-05,  1.0000e+00,  6.0000e-06,  1.0000e+00,  6.0000e-07,\n",
       "          1.0000e+00],\n",
       "        [ 6.5699e-01, -5.9944e-01,  6.4422e-01,  9.7560e-01,  6.9943e-02,\n",
       "          9.9976e-01,  6.9999e-03,  1.0000e+00,  7.0000e-04,  1.0000e+00,\n",
       "          7.0000e-05,  1.0000e+00,  7.0000e-06,  1.0000e+00,  7.0000e-07,\n",
       "          1.0000e+00]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 奇数列\n",
    "pe[:, 1::2] = torch.cos(pos / (10000 ** (2 * torch.arange(1, d_model, 2) / d_model)))  # 输出: torch.Size([50, 512])\n",
    "pe"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
